Soft Margin Bayes-Point-Machine Classification via Adaptive Direction Sampling

Supervised machine learning is an important building block for many applications that involve data processing and decision making. Good classifiers are trained to produce accurate predictions on a training set while also generalizing well to unseen data. To this end, Bayes-PointMachines (bpm) were proposed in the past as a generalization of margin maximizing classifiers, such as Support-Vector-Machines (svm). For bpms, the optimal classifier is defined as an expectation over an appropriately chosen posterior distribution, which can be estimated via MarkovChain-Monte-Carlo (mcmc) sampling. In this paper, we propose three improvements on the original bpm classifier. Our new statistical model is regularized based on the sample size and allows for a true soft-margin formulation without the need to hand-tune any nuisance parameters. Secondly, this model can handle multi-class problems natively. Finally, our fast adaptive mcmc sampler uses Adaptive Direction Sampling (ads) and can generate a sample from the proposed posterior with a runtime complexity quadratic in the size of the training set. Therefore, we call our new classifier the Multi-class-Soft-margin-Bayes-Point-Machine (msbpm). We have evaluated the generalization capabilities of our approach on several datasets and show that our soft-margin model significantly improves on the original bpm, especially for small training sets, and is competitive with svm classifiers. We also show that class membership probabilities generated from our model improve on Platt-scaling, a popular method to derive calibrated probabilities from maximum-margin classifiers.